FIG. 1 shows an exemplary Wide Field of View (WFOV) lens system. Such lens systems, nominally have a hemispherical field of view mapped to a planar image sensor for example as shown in FIG. 2.
It will be appreciated that this mapping results in variations in acquired image distortion and resolution across the field of view. It can be desirable to correct for this distortion so that for example, features such as faces especially those located towards the periphery of the field of view do not appear distorted when displayed.
Separately, it is appreciated that such WFOV systems especially introduce heavy and in some cases non-uniform distortion patterns across the field of view so that acquired images (or indeed different colour planes of an acquired image) do not uniformly conform to the ideal mapping shown in FIG. 2. Thus, again it can be desirable to correct for this distortion.
U.S. Pat. No. 5,508,734 discloses a WFOV lens assembly designed to optimize the peripheral regions of the field of view to provide improved resolution matching between the peripheral region relative to a central region, the peripheral region tending to have a lower resolution than the central region.
Referring to FIG. 3, applications such as PCT/EP2011/052970 (Our Ref: P100750pc00/FN-353-PCT) and U.S. application Ser. No. 13/077,891 (Ref: FN-369A-US) disclose digital image acquisition devices including WFOV lens systems. Here, distorted WFOV images are read from a sensor via an imaging pipeline which can carry out simple pre-processing of an image, before being read across a system bus into system memory.
Such systems can employ hardware modules or sub-modules also connected directly or indirectly to the system bus for reading successive images stored in system memory from the bus and for processing the image before either returning the processed image to system memory or forwarding the processed image for further processing. In FIG. 3, for example, a WFOV correction module successively reads distorted images or image portions and provides corrected images or image portions to a face detection (FD) and tracking module.
A system controller controls the various hardware modules, the system controller being responsive to, for example, commands received through a control interface from, for example, software applications running on the device with which a user interacts. In FIG. 3, a zoom and pan module is connected to the controller and this in turn communicates with the WFOV correction module to determine which part of an acquired image needs to be read from system memory for correction and for example, display on the device viewfinder (not shown) and/or forwarding to the face detection module. In this case, a mixer module, for example, superimposes boundaries around faces which have been detected/tracked for display on the device viewfinder.
US 2010/0111440, Chai discloses a distortion correction module which partitions coordinate points in a selected output image into tiles. The output image is an undistorted rendition of a subset of the lens-distorted image. Coordinate points on a border of the tiles in the output image are selected. For each tile, coordinate points in the lens-distorted image corresponding to each selected coordinate point in the output image are calculated. In addition, for each tile, a bounding box on the lens-distorted image is selected. The bounding box includes the calculated coordinates in the lens-distorted image. The bounding boxes are expanded so that they encompass all coordinate points in the lens-distorted image that map to all coordinate points in their respective corresponding tiles. Output pixel values are generated for each tile from pixel values in their corresponding expanded bounding boxes.
In modern high definition image acquisition devices, enormous amounts of information are received and transmitted across the system bus at high frame acquisition speeds. This places pressure on the many processing modules, such as the correction modules of FIG. 3 and Chai which are connected to the system bus to ensure their demands on the system bus are within an allocated budget and so do not interfere with other processing, but also that the processing modules themselves are implemented with the minimal hardware footprint so as to minimize device production costs.
Part of any correction module footprint is cache memory. On the one hand it is desirable to minimize cache size to minimize device cost, yet on the other hand, it is desirable to minimize I/O access by hardware modules across the system bus. So for example, where multiple forms of distortion are to be corrected, it would not be possible or acceptable to successively read from, correct and write back to memory an image for each form of distortion to be corrected.
Separately, it will be appreciated that WFOV lens systems as well as being incorporated into hand-held digital image acquisition devices can be included in devices with various specialist applications, for example, fixed security cameras. In some cases, for example, an overhead camera mounted towards a centre of a ceiling in a room might have a lens system which primarily emphasizes the circumferential field of view of the room and acquires relatively little detail in the region immediately below the camera.
When a person walks across such a room they move closer to the camera, but the angle of incidence of their face to the camera means the camera view of their face becomes less frontal possibly making it more difficult for the camera to track and/or recognise the person's face. In a case such as this, as well as correcting for the distortion introduced by the non-linear mapping of the circumferential view of the room onto the planar surface of the acquisition system imaging sensor, it may be desirable to adjust either the sensor, or lens angle to improve the view of a target person (clearly involving some loss resolution in other regions of the field of view).
Depending on the nature of the lens assembly, it may be preferable to tilt the lens, rather than the sensor. However, if the lens is a large optical assembly, for example, for providing long-range optical quality for security applications, then it could also be desirable to tilt the image sensor assembly, as indicated by the arrows of FIG. 1, to optimize the view of a person's face as they approach the camera. This tilting of the sensor introduces additional distortion into the image over that of the non-linear optical structure of the lens.
It will also be appreciated that as a person approaches the camera, their face will become elongated towards the chin and bulbous towards the top of the head. It may be thus desirable to counter this non-linear distortion of the person's face.
From the foregoing, it is clear that several different distortions occur as a person walks across the field of view (FOV) towards the lens assembly: (i) a non-linear lens distortion which can be a function of the location within the FOV of the lens; (ii) distortion due to possible relative movement of the lens and sensor surfaces; and (iii) distortion effects in local areas such as faces which vary according to both the vertical and horizontal distance from the camera unit.
Other distortions “rolling shutter” distortion and again caused by movement within the field of view while an image is being read from a sensor—thus without correcting for this distortion, portions of an image can appear wrongly shifted related to others.
In other applications, it may be desirable to flip an acquired image before it is displayed and again this can be considered as a form of distortion which needs to be corrected.
It is an object of the present invention to provide an improved correction module for a digital image acquisition device addressing the above problems.